Displays such as televisions, computer monitors, displays in portable devices, special purpose displays such as virtual reality displays, vehicle simulators, advertising displays, stadium displays, and the like are widespread. State of the art displays are capable of displaying fine nuances of color and tone.
A wide range of display technologies are now available. For example, there are plasma displays, LCD displays backlit by a variety of types of light sources such as LEDs of various types, fluorescent lamps or high-intensity incandescent lamps, CRT-based displays, digital cinema displays, OLED displays etc. A particular display combines display hardware with video signal processing components that receive video signals and drive display hardware to display video content of the video signals.
Different displays may vary significantly with respect to features such as:
the color gamut that can be reproduced by the display;
the maximum brightness achievable;
contrast ratio;
resolution;
acceptable input signal formats;
color depth;
white level;
black level;
white point;
grey steps;
etc.
Because displays can perform very differently, the same image content may appear different when viewed on different displays. Image content that is pleasing to look at when viewed on one display may be less pleasing when viewed on another display having different capabilities. Image content that matches a creator's creative intent when displayed on some displays may depart from the creator's creative intent in one or more ways when viewed on other displays.
The perception of color and luminance is also affected by ambient conditions. Video or other images presented under theater conditions (low ambient lighting) may be perceived by viewers significantly differently than the same video or other images would be perceived when viewed under conditions with significant ambient light. Further, the characteristics (such as the color temperature) of ambient light can affect a viewer's perception of video content.
The creator of a video production or other image may set tones and colors of pixels in the image so that, when viewed, the image has a desired appearance which agrees with the creator's creative intent. For example, a creator may wish some scenes to have a darker, more oppressive, feel than others. The creator may wish certain features depicted in a scene to stand out or to be less prominent. The creator may wish to have colors seem extra vivid in some scenes and more muted in others. Adjusting tones and colors of pixels in an image may include performing color grading (or ‘color timing’) on the source video data. Color grading may be performed using a hardware/software system that permits a user to change the video data in various ways to achieve a desired appearance. Color grading of an entire sequence of image frames or video therefore generally is a lengthy and costly process.
Since the choice of display on which content is viewed and the ambient lighting conditions at the time the content is viewed can affect viewers' perceptions of the content being viewed, it would be ideal to have the creator perform color grading separately for every display on which the video production might be viewed and for all ambient conditions under which the video production might be viewed. Viewers could then obtain customized versions of the production optimized for viewing on their displays and ambient conditions. This is generally impractical.
Commonly owned U.S. patent application No. 61/307,547 filed on 24 Feb. 2010 and entitled DISPLAY MANAGEMENT METHODS AND APPARATUS and No. 61/366,899 entitled DISPLAY MANAGEMENT SERVER and No. 61/364,693 filed on 10 Jul. 2010 and entitled DISPLAY MANAGEMENT METHODS AND APPARATUS describe generating video content having colors and tones suitable for display on a particular display by interpolating or extrapolating between different version of the video content that have been prepared for viewing on other displays. Further commonly owned U.S. patent applications No. 61/453,924 filed on 17 Mar. 2011 and entitled GENERATING ALTERNATIVE VERSIONS OF IMAGE CONTENT USING HISTOGRAMS as well as No. 61/453,922 filed on 17 Mar. 2011 and entitled ENCODING AND DECODING ALTERNATIVE VERSIONS OF IMAGE CONTENT USING HISTOGRAMS describe various applications of a progressive histogram matching algorithm to match the visual perception of at least one source image to the desired visual perception of at least one target image. These applications are hereby incorporated herein by reference for all purposes in their entirety.
For example, a source image may be color graded for viewing on a REC 709 display and another source image may be color graded for viewing on a VDR display. So usually the visual perception of such source images will differ widely when viewed on the same target display. If the target display is neither one of the target displays to which said source images have been specifically adapted, both source images' on screen appearance will usually not cause excitement. Furthermore, the intended creative impression to be conveyed by the source images' appearance on screen might be totally lost or at least considerably reduced.
Rec 709 is a video data format specified by ITU-R Recommendation BT.709, which is hereby incorporated herein by reference. Visual Dynamic Range (VDR) is a format capable of representing an extremely broad range of colors and tones. VDR is described, for example, in co-owned PCT Application No. PCT/US2010/022700 entitled EXTENDED DYNAMIC RANGE AND EXTENDED DIMENSIONALITY IMAGE SIGNAL CONVERSION AND/OR DELIVERY VIA LEGACY VIDEO INTERFACES which is hereby incorporated herein by reference for all purposes. The VDR format can encode the full range of human vision.
There is a need for efficient ways to achieve a satisfying viewing quality on one or more specific target displays while still preserving the original creative intent embodied in e.g. the coloring of the image content to be viewed. Approaches that have been used to achieve the same are described in the references listed in this Background section, below.
Pitié et al. “Automated colour grading using colour distribution transfer,” Computer Vision and Image Understanding, 107(1-2):123-137, 2007 as well as Pitié et al., “N-dimensional probability density function transfer and its application to colour transfer,” IEEE International Conference on Computer Vision (ICCV'05), Beijing, China, 17-21 October, 2, 2005, 1434, 1439 focus largely on full color transfer, which works for images that may differ significantly from each other in a spatial sense and/or in a sense that relates to brightness and color.
Reinhard et al., “Color Transfer between Images” (IEEE CG&A special issue on Applied perception, Vol 21, No 5, pp 34-41, September-October 2001 as well as Pouli et al., “Progressive Histogram Reshaping for Creative Color Transfer and Tone Reproduction” (Proceedings of the ACM Symposium on Non-Photorealistic Animation and Rendering, Annecy, France, June 7-10, pp. 81-90, 2010) describe basic algorithms to transfer the “mood” or “intent” of one image to another while not altering its structure. Hereby, the Pouli et al. reference specifically teaches progressive histogram matching methods to partially match a source to a target image. In chapters 3 and 4 (entitled “Progressive Histogram Reshaping”, respectively “Creative Tone Reproduction”) the algorithm is laid out in detail. Such chapters 3 and 4 shall be explicitly incorporated herein by reference in their entirety. In a later paper, Pouli et al. present a further elaborated amended version: “Progressive color transfer for images of arbitrary dynamic range”, Computer & Graphics 35 (2011) 67-80 of such method. In chapters 3, 4 and 5 (entitled “Algorithm”, “Region Selection” respectively “Creative tone reproduction”) on pages 69 up to 76, the method is outlined in detail. Such chapters 3-5 shall be explicitly incorporated herein by reference in their entirety.